Skip to content

feat: support geospatial results on the kernel path - #990

Merged
cathleeny merged 14 commits into
mainfrom
feat/geospatial-results
Oct 19, 2026
Merged

cathleeny merged 14 commits into
mainfrom
feat/geospatial-results

Conversation

@cathleeny

@cathleeny cathleeny commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • add the kernel-only, default-disabled enable_geospatial_support connection option
  • return canonical {"srid": int, "wkb": bytes} values when enabled and WKT / EWKT strings when disabled
  • preserve logical GEOMETRY / GEOGRAPHY Arrow metadata
  • pass the value through the typed PyO3 argument only; it is never forwarded to SEA
  • align table-type E2E coverage with the case-sensitive filtering from kernel #393, included in the new kernel pin; assert uppercase matches and lowercase/mixed-case non-matches

Dependency

Pins kernel 9c2e2e6182fc7285bcfff39b2177941bc804897c, including kernel #361. Before a connector release, #991 must land to raise the optional published-kernel dependency floor from ^1.1.0 to ^1.2.0.

Validation

  • unit suite: 1,263 passed, 363 subtests passed, 4 skipped, 2 deselected
  • real-kernel signature/routing tests against the exact kernel extension: 2 passed
  • live staging Reyden and DBSQL E2E: all 48 cases passed in omitted/default-disabled, explicit-disabled, and explicit-enabled modes through row/Arrow APIs and both complex-value modes; all 150,007 large-result rows checked
  • Thrift compatibility: four complex-JSON cases fail identically before and after this change; native-complex mode passes

Merge-fix validation

  • Fix commit: 3c800a507b23817cc764aa7c87964dda5a5e71e1.
  • Code quality: all 31 unit-test, lint, and type-check jobs passed on the final PR head.
  • Merge-queue Kernel E2E: 45 backend tests and 3 TLS tests passed, including test_tables_table_types_filter_is_case_sensitive; proxy traffic verification passed.
  • Merge-queue integration: both Thrift and kernel integration suites passed.
  • Merge-queue coverage: 1,601 passed, 7 skipped, and 363 subtests passed; coverage enforcement passed.

Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium

Solid, well-tested feature (kernel geospatial passthrough + thrift description logical-type derivation). One medium concern: the default-True option hard-fails every kernel session against a wheel that lacks the kwarg, while the pip constraint (^1.1.0) still permits such wheels — worth gating the default case or bumping the minimum in lockstep. The thrift positional-field switch and the variant/geometry/geography regex both look correct and are covered by new tests.

Comment thread src/databricks/sql/backend/kernel/client.py
@cathleeny
cathleeny disabled auto-merge October 19, 2026 20:29
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium

Solid, well-tested change wiring a kernel-only enable_geospatial_support option and generalizing the Thrift logical-type extraction (variant → variant/geometry/geography, positional field lookup to handle duplicate aliases). One medium documentation mismatch: CONNECTION_PARAMETERS.md lists the default as True while code, docstring, CHANGELOG, and tests all default it to False. The intentional always-explicit _kernel_geospatial_kwargs (which hard-fails old kernel wheels even at the default) is clearly deliberate and covered by tests.

Comment thread CONNECTION_PARAMETERS.md Outdated
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium

Solid, well-tested feature. One medium concern worth maintainer confirmation: the default-False geospatial path raises NotSupportedError on older kernel wheels, unlike every other optional kwarg which degrades gracefully — so a connector upgrade hard-breaks all kernel connections against an older wheel even for users who never enable geospatial. The thrift _col_to_description regex and positional-index schema-lookup changes look correct and are well-covered by the new tests.

Comment thread src/databricks/sql/backend/kernel/client.py
@cathleeny
cathleeny added this pull request to the merge queue Oct 19, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 19, 2026
@cathleeny
cathleeny added this pull request to the merge queue Oct 19, 2026
Align Kernel E2E with case-sensitive filtering from databricks-sql-kernel#393. Check uppercase matches and lowercase/mixed-case empty results, and update stale filter comments.

Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 19, 2026
@cathleeny
cathleeny enabled auto-merge October 19, 2026 22:24
@cathleeny cathleeny added the kernel-e2e Trigger preview run of the Kernel E2E workflow on this PR label Oct 19, 2026

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 High

One high-severity backward-compat concern: the enable_geospatial_support kwarg is sent to the kernel unconditionally and hard-fails against any ^1.1.0-permitted kernel wheel lacking it, even at the default False — breaking existing default kernel connections after a connector-only upgrade. The Thrift _col_to_description regex, positional _hive_schema_to_description lookup, and the result-set/description tests otherwise look correct and well-covered.

Comment thread src/databricks/sql/backend/kernel/client.py
Refresh status contexts after GitHub reported the Kernel E2E requirement as expected despite matching successful checks. The source tree is unchanged from the validated fix in 3c800a5.

Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
@github-actions github-actions Bot removed the kernel-e2e Trigger preview run of the Kernel E2E workflow on this PR label Oct 19, 2026

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium · 1 Low

Solid, well-tested feature addition. One medium concern: the geospatial kwarg hard-fails on default-disabled connections against older/currently-published kernel wheels (release must be gated on the #991 floor bump, or the default case should fall closed like max_connections). Plus a low note on the new positional Arrow-field mapping assuming schema order parity for all Thrift descriptions.

Comment thread src/databricks/sql/backend/kernel/client.py
Comment thread src/databricks/sql/backend/thrift_backend.py
@cathleeny
cathleeny added this pull request to the merge queue Oct 19, 2026
Merged via the queue into main with commit afc6cb8 Oct 19, 2026
54 checks passed

This branch was successfully deployed

1 active deployment
azure-prod — 8d3c3db3 Deployed Oct 19, 2026 by peco-review-bot[bot] via followup #1013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants